BlueOnyx APIv2 released

BlueOnyx now has a REST API written in Golang with full (and enhanced) access to CCEd.

BlueOnyx API (v2):

The initial BlueOnyx API (v1) had its shortcomings, therefore it has been superseded by a more modern 'cced-api' daemon written in Golang, which directly interfaces with CCEd and allows us to securely expose all CCEd functionality. Remote access is possible for whitelisted IP addresses with secrets and tokens that are only valid for the respective whitelisted IPs.

A WHMCS module to manage BlueOnyx servers is available free of charge here. This modules allows provisioning Vsites via WHMCS for both BlueOnyx 5210R and BlueOnyx 5211R and will also cover the upcoming BlueOnyx 5212R.  

API GUI management:

The API v2 is always enabled and is represented by the service "cced-api.service". This cced-api process runs as user 'admserv' and and normally binds on 127.0.0.1:9092 (TCP). This API then only responds to local POST requests. A full documentation is available here. If desired, the API can be exposed to selected whitelisted IP addresses or IP address ranges. However: Even then access to the API is only possible via a "Client-Secret", which the GUI auto-generates for each IP address or IP address range. 

Backstory - the Why and the How:

In order to function the BlueOnyx GUI needs to communicate with CCEd. Which is used for authentication, session management, ACLs, CODB database access and the running of handlers, constructors and CCEWrappers. CCEd is reachable locally via a Unix socket.

In BlueOnyx 5210R and 5211R this "interface" between the GUI and CCEd was written in PHP. But PHP calls to a Unix Socket are not really high performance, because PHP can't really "talk" and "listen" on the socket at the same time. It needs to do context switching and the pre- and post-processing of calls to CCEd also takes up considerable CPU cycles and code cruft.

I recently observed that more and more projects are utilizing Golang and it became quickly obvious why. Golang is essentially a scripting language, too. It's syntax is easily approachable and the resulting scripts can (if need be) run via the "go" interpreter. But the typical usage case is to use Golang and let it compile the script(s) into a binary, which then executes with the speed and usual benefits of a traditional binary.

Golang is also very modular and has modules for many purposes. Need it to include a small webserver for an API? Load the module for that. Need encryption? Socket access? Regular expression handling and Syscalls? Just load the modules. At the end of the day the whole cced-api.go source code is just 2000 lines of comfortably spaced code with comments. Which is surprisingly little for what it does.

Another benefit of cced-api is: It allowed us to introduce new commands to the CCEd toolbox. Commands that combine several native CCEd commands into higher level and more functional commands. Such as GETALL or GETOBJECT. Replicating these in just PHP to execute the underlying FIND and GET requests is slow and tedious, but when cced-api does them for us? It can run these at full speed of a dedicated binary, which then greatly speeds up the GUI for those requests that utilize these newer commands.

How it's used:

The GUI now uses cced-api by default when it accesses CCEd. This speeds up most transactions and makes the GUI feel a little bit more "snappy". If cced-api is down or unresponsive for whatever reason? The GUI will detect it and will remedy the situation. And in the meantime it will perform a fallback to the traditional CCE.php based direct Unix socket access to CCEd. 

Likewise: If remote API access is enabled, then you have full (and secure) access to CCEd from remote and can poll CCEd for information and can perform transactions without using the GUI. This allows for remote provision, monitoring and a more seamless integration of BlueOnyx into third party infrastructure.

To that end we have already written a module for WHMCS which allows remote provisioning of Vsites and users accounts from within the WHMCS portal system. A module for Fossbilling (a free and open source solution similar to WHMCS) is planned.

Security:

Yes. Present and accounted for.

Like said above: By default cced-api is only reachable from localhost on the BlueOnyx and it uses HTTPS POST requests. A local user (with shell or script access) can attempt to communicate with cced-api, but even when he does? At the most he can login with his own username and password to cced-api and then can only perform the same transactions he can perform via the GUI. Beyond that no information not visible to the user via the GUI is exposed or read- or write-able.

Remote access (if enabled for cced-api) requires that the remotely accessing IP is whitelisted. Beyond that a user from such a whitelisted IP must also know the "Client-Secret" associated with the whitelisted IP and that user has to request a short lived token that's only valid within that particular session context.

Future Plans:

The introduction of cced-api and switching the GUI to use it? That's only the first step and we won't stop there. 

Long term? CCEd itself will also be rewritten in Golang to facilitate easier code maintenance and more modern methods than currently employed. If rewritten in Golang we can easily make CCEd extensible. Tack on different databases than the current CODB, add database indexing (a direly missed feature at the moment) and can merge currently externalized functionality directly into CCEd for better performance.

Once CCEd has been rewritten in Golang we can also modernize "cceclient", the shell based terminal to manage CCEd from the CLI. Make it more useful for routine maintenance and diagnosis. Even let it access remote BlueOnyx servers via the API. Even a version of cceclient is imaginable that runs on your local Linux workstation or under Windows. Because Golang binaries can be compiled statically, so they are more or less independent of what flavor of OS you're using. That's a bit of a simplification, but yeah: It's doable.

Lastly: "One GUI to rule them all!" If CCEd is written in Golang and cced-api is properly configured and exposed for secure remote access? Then long term it will be possible to use the GUI of one BlueOnyx to manage all API connected other BlueOnyx servers under your control.

There are also other interesting usage cases. Such as letting Easy-Migrate and/or Easy-Backup use the API to perform migrations and backups. Because beyond just sending and receiving JSON data via POST requests we can establish data-streams via the API to handle exports and imports. All within the security context of the GUI and the API itself: Only for authorized users after proper authentication and if their privilege level is sufficient for such transactions. 

In the longer haul the API will allow BlueOnyx to learn new tricks and to do its work much more efficiently.